Question difficulty poses a serious threat to the reliability and validity of survey data. However, very little is known about the factors that determine question difficulty. Only recently have survey researchers started to look at specific text features in order to explain why some questions are easier to comprehend or have less cognitive burden than others. Theoretical and empirical evidence from psycholinguistics suggests that these features (e.g. low-frequency words, vague noun-phrases, left-embedded syntax) cause comprehension difficulties and can thus have a strong impact on survey response quality. In order to examine the effects of these text features on question difficulty, we conducted an online experiment in which two versions of the same questions were compared using response latencies. Response latencies are assumed to reflect the cognitive effort that is required to answer a survey question with poorly formulated questions resulting in longer response times. One group (n = 495) received well-formulated survey questions, the other group (n = 490) answered questions which were suboptimal with respect to several psycholinguistic text features. Consistent with our predictions, the results reveal a significant difference between both conditions for most questions. Moreover, the answer distributions of several questions differed significantly between conditions, a finding which is quite alarming given that the question alternatives are supposed to measure the same concept. Hence, questionnaire designers are advised to pay attention to these text features when crafting questions.